What is Google-Extended?
Google-Extended is a robots.txt token, not a crawler, and it has no user agent string of its own. Publishers use it to manage whether content Google crawls may train future Gemini models or ground answers in Gemini Apps. Google says it does not affect inclusion in Google Search or ranking. Verified 2026-11-06.
By SeenLens · Last updated
Free, no sign-up. We read robots.txt and request the page with a browser and four crawler user agents. We do not store your page text.
Google-Extended fact sheet
| Fact | Value |
|---|---|
| Operator | |
| Purpose | Control token. It manages whether content Google crawls may be used for training future Gemini models (Gemini Apps and the Vertex AI API for Gemini) and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. |
| User agent string | None. Google: "Google-Extended doesn't have a separate HTTP request user agent string." Crawling uses existing Google user agent strings. |
| robots.txt token | Google-Extended |
| Published IP list | None for Google-Extended itself, because it does not crawl. Google's common crawlers list is common-crawlers.json. |
| Obeys robots.txt | Google says its common crawlers always obey robots.txt rules when crawling automatically. The Google-Extended token is how robots.txt controls the AI uses above. |
| Related bots | Googlebot (Search), Google-CloudVertexBot (Vertex AI Agents), Applebot-Extended (Apple's similar token) |
| Last verified | 2026-11-06 |
What does Google-Extended do?
Google-Extended is a switch in robots.txt. It does not fetch pages. Google fetches pages with its normal crawlers, and the Google-Extended token tells Google how it may use what it fetched.
Google lists two uses that this token controls. The first is training of future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini. The second is grounding, which Google defines as providing content from the Google Search index to the model at prompt time to improve factuality and relevancy.
Google says it intends to support the AIPREF shared machine-readable training preference signal (user-agent: __usage_ai-training) in early 2027.
How do I block Google-Extended in robots.txt?
To opt out of the uses above for the whole site:
User-agent: Google-Extended
Disallow: /Google's own example blocks a folder and allows one path inside it:
user-agent: Google-Extended
allow: /archive/1Q84
disallow: /archive/This file keeps Googlebot crawling for Search and blocks Google-Extended:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /Does Google-Extended affect AI Overviews or Google Search?
Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.
For AI Overviews, Google points elsewhere. In its AI features documentation, Google says AI is built into Search, so robots.txt directives for Googlebot are the control for how your site is crawled for Search. To limit the information shown from your pages in Search, Google lists nosnippet, data-nosnippet, max-snippet and noindex. It names Google-Extended only for limiting AI training and grounding in some of Google's other systems.
Blocking Google-Extended therefore does not remove your site from AI Overviews. Google's documentation gives snippet controls and Googlebot rules for that.
How do I check whether my site blocks Google-Extended?
Use the AI crawler checker above, preselected for Google-Extended. It reads your robots.txt and shows the group and rule line that match your page path. It does not send a request as Google-Extended, because the token has no user agent and no fetch of its own.
No log search can find Google-Extended either. Look for Googlebot in your logs to confirm that Search crawling still works.
What are the common Google-Extended mistakes?
- Blocking Googlebot to opt out of Gemini training. That removes your pages from Search crawling. Block the
Google-Extendedtoken instead. - Blocking Google-Extended to leave AI Overviews. Google says its AI features follow Googlebot and snippet controls.
- Looking for a Google-Extended user agent in logs. There is none. Google says the token has no separate HTTP request user agent string.
- A wildcard group.
User-agent: *withDisallow: /blocks Googlebot as well as Google-Extended unless Googlebot has its own group. See robots.txt blocks AI crawlers. - A firewall rule that treats Google traffic as a bot. Google-Extended is not fetched, but Googlebot is. If Search crawling fails, see AI crawlers blocked by a firewall.
Google-Extended FAQ
Is Google-Extended a crawler?
No. Google calls it a standalone product token. It has no separate HTTP request user agent string, and Google crawls with its existing user agents.
Does Google-Extended affect AI Overviews?
Google says Google-Extended does not impact inclusion in Google Search. Its AI features documentation names Googlebot rules and snippet controls as the Search controls.
What does Google-Extended control?
Whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
How do I block Google-Extended?
Add User-agent: Google-Extended and Disallow: / to your robots.txt.
Will blocking Google-Extended lower my Google rankings?
Google says Google-Extended is not used as a ranking signal in Google Search.
Sources
- Google: Google's common crawlers (token definition, no user agent string, products affected, effect on Search, AIPREF note, Google-CloudVertexBot, IP list; checked 2026-11-06)
- Google: AI features and your website (Googlebot and snippet controls for AI features in Search; checked 2026-11-06)