Crawler reference / robots.txt
Google-Extended in robots.txt and what it controls
Google-Extended has no HTTP user agent string of its own, so the name never shows up in a server log. It is a control token that Google reads from your robots.txt. Below are Google's own words, a table of which Google control does which job, and four robots.txt files with the group that each token follows.
What Google says about the token
Google's crawler page (last updated 2026-10-30) gives three facts.
- It has no separate HTTP user agent string. Google writes: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." The fetches carry the user agent strings of Google's other crawlers, so you cannot pick out or block Google-Extended by user agent or by IP address. The reverse and forward DNS check confirms that a request comes from Google. It cannot tell you that a request is for Google-Extended.
- It covers Gemini training and grounding. Google writes that publishers can use it "to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI."
- It does not touch Search. Google writes: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
Google announced the token on September 28, 2023 as a control for "Bard and Vertex AI generative APIs". The current crawler page names Gemini instead.
Which Google control does which job
| Control | What it governs, in Google's words | Effect on Google Search |
|---|---|---|
Googlebot (robots.txt) | "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search", and AI is "built into Search" | Controls crawling for Search, AI features in Search included |
Google-Extended (robots.txt) | "training future generations of Gemini models" and "grounding" in "Gemini Apps and Grounding with Google Search on Vertex AI" | None. Google says it "does not impact a site's inclusion in Google Search" |
Google-CloudVertexBot (robots.txt) | Crawls "requested by the site owners' for building Vertex AI Agents" | None. Google says "It has no effect on Google Search or other products." |
nosnippet, data-nosnippet, max-snippet, noindex (page level) | "To limit the information shown from your pages in Search" | Controls what Search shows from the page, AI features included |
Google's crawler page, checked 2026-11-08, also says that Google "intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: __usage_ai-training") in early 2027." This is a plan, not a control that works today.
Which group Google-Extended follows
Two rules decide this. RFC 9309 says: "Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group." and "If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present." Google's robots.txt page adds: "User agent specific groups and global groups (*) are not combined." The four files below show the result for the Google-Extended token and the Googlebot token.
File 1 has only a star group.
User-agent: *
Disallow: /private/
Both tokens follow the star group. For both, /private/ is disallowed and the rest of the site is allowed.
File 2 adds a Google-Extended group that disallows the whole site.
User-agent: *
Disallow: /private/
User-agent: Google-Extended
Disallow: /
Google-Extended follows its own group, so no URL on the site may be used for Gemini training or grounding. Googlebot has no group of its own, so it follows the star group and Search crawling is unchanged.
File 3 has a Google-Extended group that disallows one folder only.
User-agent: *
Disallow: /private/
User-agent: Google-Extended
Disallow: /drafts/
Google-Extended follows only its own group. The star rules do not apply to it, because specific groups and the star group are not combined. So /drafts/ is blocked for Google-Extended, but /private/ is not. To keep both folders out, repeat Disallow: /private/ in the Google-Extended group.
File 4 writes the token in lower case.
User-agent: *
Disallow: /private/
user-agent: google-extended
Disallow: /
The result is the same as file 2. Google writes: "Both the user-agent field name and its value are case-insensitive." The path values are not: Google writes that the value of disallow "is case-sensitive".
How to check the result
Test the live file in the robots.txt checker with the token google-extended, and compare the group it selects with the examples above. Do not look for Google-Extended in your access logs. It has no HTTP user agent string, so the name never appears there. The OpenAI crawlers page and the Claude crawlers page cover the same questions for those vendors.
Sources
- Google: Google's common crawlers (Google-Extended and Google-CloudVertexBot entries, last updated 2026-10-30)
- Google Search Central: AI features and your website
- Google: How Google interprets the robots.txt specification
- RFC 9309, section 2.2.1
- Google: Verify requests from Google crawlers and fetchers
- Google blog: An update on web publisher controls (September 28, 2023)
Checked 2026-11-08 against the Google pages above. Vendor wording can change; re-read the source before you copy a rule.
