Loading
AI crawler comparison
Neither is Googlebot, and only one of them is an AI training control.
Short answer
They are not interchangeable. Google-Extended handles AI model training, GoogleOther handles both training and AI search. Blocking one has no effect on the other, so a robots.txt rule aimed at the wrong agent produces exactly the outcome you were trying to avoid.
| Google-Extended | GoogleOther | |
|---|---|---|
| Operator | ||
| Purpose | AI model training | Training + search |
| robots.txt | Respects robots.txt | Respects robots.txt |
| Published IP ranges | Yes — verifiable | Yes — verifiable |
| Official docs | Yes | Yes |
| Blocking costs you | No traffic — training only | AI-answer citations |
Google-Extended is not a separate crawler — it is a robots.txt control token evaluated by Google's existing crawl infrastructure. Disallowing it opts your content out of Gemini training and grounding without affecting Google Search indexing or ranking, which remain governed by Googlebot.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: Google-Extended.
GoogleOther is Google's catch-all crawler for uses outside Search indexing — research, internal development, and product one-offs, which can include AI-related work. It exists so those fetches don't masquerade as Googlebot. Blocking it does not affect Google Search rankings, which remain governed by Googlebot.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: GoogleOther.
User-agent: Google-Extended Disallow: / User-agent: GoogleOther Disallow: /
The split most sites want when the two agents do different jobs: opt out of one without giving up the other.
User-agent: Google-Extended Disallow: / User-agent: GoogleOther Allow: /
Google-Extended is Google's AI model training agent — Controls use of content for Gemini and Vertex AI training, separate from Googlebot. GoogleOther is Google's both training and AI search agent — Google's generic crawler for research and internal product uses. They are separate user agents with separate robots.txt rules.
No. robots.txt matches on the user-agent token, so a group naming Google-Extended applies only to Google-Extended. GoogleOther reads the group that names it, or the wildcard group if none does. To stop both you need a rule for each — or a wildcard rule, which would also affect every other crawler that reads it.
It depends which outcome you want. Blocking Google-Extended opts you out of Google's model training and costs you no traffic today, because training crawlers send no visitors. Blocking GoogleOther removes you from the answers Google's assistant shows its users. The common choice is to block training agents and allow search agents, so your content stays citable without feeding model training.
Google-Extended: As a robots.txt token rather than a crawler, honoring your directive is its entire function; Google documents it officially. GoogleOther: Officially documented by Google and respects robots.txt like the rest of its crawler family.
Yes — they send different user-agent strings, so grep for each token separately: grep -i "Google-Extended" and grep -i "GoogleOther". Be aware that a user-agent header is self-declared and anything can send either string. For a check that cannot be spoofed, match the request IP against the published ranges: Google-Extended at https://developers.google.com/static/crawling/ipranges/common-crawlers.json, GoogleOther at https://developers.google.com/static/crawling/ipranges/common-crawlers.json.
Part of our directory of every known AI crawler. Last verified: 2026-09-13.