On October 30, Google and Microsoft announced support for a shared robots.txt token, __usage_ai-training, to opt out of AI training across several providers in a single line, building on a draft filed the same day at the IETF by an Anthropic engineer. OpenAI announced it would honor the shared signal the same day. The token only covers training: it does not affect Googlebot's crawling, nor, at Microsoft, the appearance of a site in generated answers, which is controlled separately. Google is aiming for early 2027, Microsoft sets February 1, 2027, and a March 1, 2027 deadline for sites still using NoArchive and NoCache. For a small business, nothing forces action before the rollout, but the setting is worth preparing.
Contents
Google announced on October 30, 2026 that it plans to support a shared preference token in robots.txt, built on a draft from the AIPREF working group at the Internet Engineering Task Force (IETF), so sites can express their AI training preferences to several providers in a single step.1 Microsoft published its own controls around this shared token the same day, and OpenAI announced it would honor the shared signal; Anthropic, which filed the draft at the IETF, is also taking part in the discussions, according to PPC Land.2 This article covers what the token does, what it leaves out, the timeline announced by each provider, and the setting worth considering for a small business that wants to keep being cited by AI systems.
01How the shared token works
The new mechanism relies on a policy token, written in a User-Agent line of robots.txt like any other bot, but it does not express a crawling rule: it expresses a usage preference on content that has already been retrieved.3
The syntax: one token, several providers
The IETF draft defines the token __usage_ai-training, written in an ordinary robots.txt group. The draft's own example disallows training on an entire site, except under a chosen path:3
| Goal | Robots.txt lines |
|---|---|
| Disallow training on the whole site, except one path | User-agent: __usage_ai-trainingDisallow: /Allow: /press-kit/ |
Before this token, a site that wanted to opt out of training with several providers had to repeat the rule for each one: Google-Extended, GPTBot, ClaudeBot, and others, each with its own name and its own documentation.2 The shared token replaces those repeated lines with a single one that several providers commit to reading the same way.
The safety net for Google-Extended
Google clarified a detail for sites that already have Google-Extended configured: if a site uses the new token to opt out of training without explicitly mentioning Google-Extended, Google plans to treat that as if the site had disallowed Google-Extended.1 Existing company-specific controls keep working alongside the new shared token: a site can keep a broad setting with the shared token while tailoring its choice for one specific provider through that provider's own control, such as the robot-by-robot settings described in the Generative Engine Optimization guide.1
02What the token leaves out
The IETF draft and Google and Microsoft's existing controls draw a clear line between three separate uses: crawling a site, training AI models, and appearing in a generated answer. The new token only covers the second one.3
Crawling and indexing stay the same
The IETF draft states that this policy token is not a directive for crawling behavior, and that it stays independent from the crawling rules tied to each bot's own name: a crawler still applies the rules written for its own user agent, exactly as before.3 On Google's side, its crawler documentation states that Google-Extended, the control it has operated since 2023 for training Gemini models, has no effect on a site's inclusion in Google Search or on its ranking.4 The shared token, designed to work like Google-Extended, inherits that same limit: it does not touch Googlebot or indexing.
At Google, grounding in Gemini stays a separate setting
Google's documentation states that Google-Extended covers both training future Gemini models and grounding, meaning providing content from the Search index to the model at prompt time, for responses in Gemini Apps and Vertex AI.4 Sites that specifically allow Google-Extended remain eligible to appear in Gemini Apps' grounded responses and to receive clicks from them.1 A site that wants to keep being cited in Gemini's grounded responses while opting out of training still needs to keep configuring Google-Extended specifically; the shared token alone does not separate the two uses at Google.
At Microsoft, training and grounding are set separately
Microsoft goes further by explicitly separating the two uses. Its documentation distinguishes a training preference (AI Training), set in robots.txt with the shared token or at the page level, from a grounding preference (AI Grounding), set only at the page level through a meta tag or an X-Robots-Tag header, with no sitewide equivalent in robots.txt.5 A site can disallow training while still letting Copilot cite its pages, or the other way around; neither choice changes its eligibility for ordinary Bing search results.5
03Timeline and setup for a small business
The four providers are not moving at the same pace, and only Microsoft has set a firm deadline paired with a cutoff date for sites still using its older tags.5
The announced timeline
| Provider | Announced timeline |
|---|---|
| aims to support the token during 2027; its crawler documentation says early 202714 | |
| Microsoft (Bing) | honors the token and its own training and grounding preferences starting February 1, 2027, as it crawls5 |
| OpenAI | announced on October 30 that it would honor the shared signal2 |
| Anthropic | filed the draft at the IETF on October 30 and is taking part in the working group's discussions32 |
The draft remains an individual Internet-Draft at the IETF, with no formal standing in the standards process: its wording, scope and syntax can still change before it becomes a standard.32
The firm deadline: sites under NoArchive or NoCache at Microsoft
Microsoft sets a precise timeline for sites still using its older NoArchive and NoCache tags, in place since 2023 to control training and grounding. Temporary transition preferences open in Bing Webmaster Tools on December 15, 2026, for affected sites only. The new preferences take effect as Bing crawls, starting February 1, 2027. Sites that want to keep their restrictions must publish their new preferences and, for grounding, record a temporary preference in Bing Webmaster Tools, before March 1, 2027: after that date, NoArchive and NoCache are no longer read as a training or grounding preference. The temporary preferences themselves expire on September 30, 2027.5
The setting worth considering for a small business
For a small business with no AI-specific control in its robots.txt yet, nothing requires acting before the rollout each provider announced: the current, provider-specific controls stay valid in the meantime.1 To get ready, the AI crawler access test shows, bot by bot, what the current file allows, and the robots.txt generator writes the shared token's lines once the choice is made. For a site that wants to keep being cited by Gemini or Copilot, the choice is less a single switch than a provider-by-provider decision: disallowing training with the shared token does not stop a site, at either Google or Microsoft, from keeping a separate setting to appear in grounded answers.