Robots.txt AI Training Signal: Four AI Firms Back One Opt-Out
Google, OpenAI, Microsoft and Anthropic back a shared robots.txt token for AI training opt-outs. What __usage_ai-training does and when each one applies it.
PromptWises is reader-supported. Some links may earn us a commission at no extra cost to you. This never affects our ratings or recommendations.
Google and OpenAI said on October 30, 2026 that they will honor a single robots.txt signal for AI training preferences, and Microsoft and Anthropic made the same commitment that day. Instead of one rule per AI company, a site owner adds a group for a shared policy token, __usage_ai-training, and disallows the paths they do not want used for training. Support is still arriving: Google aims for 2027, Microsoft starts on February 1, 2027, and Anthropic says it has begun recognizing the signal and is rolling it out. Sources: Google Search Central blog, the IETF draft and OpenAI’s announcement.
Update (18 November 2026): Our first version said Google and OpenAI were the only vendors with public commitments. That was wrong: Microsoft’s Bing blog and Anthropic’s crawler help page both committed on October 30. Separately, the draft was revised to -01 on November 16; only the acknowledgements changed, not the mechanism.
What the shared AI training control is
The mechanism comes from the draft “Machine-Readable Training Preference Signal” (draft-kelley-aipref-training-preference-signal), discussed in the IETF’s AI Preferences (AIPREF) working group. It adds no new directives. You write a normal robots.txt group, but the User-agent line names a policy token rather than a crawler:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
According to the draft, this expresses a preference against AI training for every path except /press-kit/. The rules that matter:
- Wildcard groups do not count. A
User-agent: *group says nothing about training. If no group names the token, no preference is expressed. - It is not a crawl rule. The draft says the token “is not a directive for crawling behavior”. To stop a crawler fetching pages, you still need crawler-specific groups.
- “AI training” is narrow. It means using content to change the learned parameters of a generative AI model; models used only for classification, ranking or scoring are excluded.
- Old parsers ignore it. It looks like a group for an unknown user agent, so existing parsers behave as before.
The author is Kevin Kelley of Anthropic. It is an individual draft, which the IETF says has no formal standing.
How Google will read it
Google’s post, by Search Advocate John Mueller, says that when a site uses the token to opt out and does not name Google-Extended explicitly, “Google plans to interpret that as if the publisher had disallowed Google-Extended”, its AI training control since 2023. Its crawler documentation says support is intended for “early 2027” and that Google-Extended does not affect inclusion or ranking in Google Search.
- Company-specific rules still work. You can opt out broadly and still allow one provider.
- There is a Gemini trade-off. Sites that specifically allow Google-Extended are eligible to appear in, and receive clicks via, grounding in Gemini Apps.
Google also cites the EU’s Code of Practice for general-purpose AI model providers, which recognizes robots.txt as a main way to state machine-readable preferences.
What OpenAI, Microsoft and Anthropic committed to
- OpenAI says it “will honor a shared robots.txt signal” for participating providers. The article returns 403 to us, so we cannot report a start date or how it interacts with GPTBot. OpenAI’s crawler docs still describe only GPTBot and OAI-SearchBot.
- Microsoft will honor the token as Bing crawls from February 1, 2027. It adds page-level training and grounding preferences (meta tag or
X-Robots-Tag), and it is replacing NoArchive and NoCache, which stop counting after March 1, 2027. - Anthropic says that as of October 30, 2026 it “has begun recognizing” the signal, rolling out across its services over time. Rules naming ClaudeBot take priority over the general setting.
What site owners should do now
- Keep your current AI rules. Google-Extended, GPTBot and ClaudeBot groups still count; Google and Anthropic say a rule naming their own token takes precedence.
- Add the token if you want a broad opt-out. Old parsers ignore it, and Anthropic says it has begun recognizing it.
- Decide by path. Think about which sections (a press kit, documentation) you would happily see used for training.
- Bing users on NoArchive or NoCache: move to the new preferences before March 1, 2027.
For how the assistants use web content, see our Gemini review, ChatGPT review and Claude review. More in the news section.
Frequently asked questions
What is the __usage_ai-training token in robots.txt?
It is a shared 'policy token' from an IETF Internet-Draft first published on October 30, 2026 and revised on November 16, 2026. You write it as the User-agent of an ordinary robots.txt group, and Disallow or Allow lines then say which paths you do or do not want used for AI training by participating providers.
Does the robots.txt AI training signal work today?
Only partly. Anthropic says it began recognizing the signal on October 30, 2026 and is rolling it out across its services. Microsoft will honor it from February 1, 2027, and Google's documentation says it intends to support it in early 2027. OpenAI has not published a start date. The specification is still an individual Internet-Draft.
Will blocking AI training with this token hurt my Google Search rankings?
Google says Google-Extended, which an opt-out under the token maps to, does not affect inclusion or ranking in Google Search, and the draft says the token is not a directive for crawling. Google does note that sites which allow Google-Extended are eligible to appear in, and get clicks from, grounding in Gemini Apps. Microsoft says its training preference does not change eligibility for Bing search results.
Which AI companies will honor the shared training opt-out?
At the time of writing, Google, OpenAI, Microsoft and Anthropic have said so in their own announcements or documentation, all on October 30, 2026. We found no other vendor commitments in primary sources.
Does the shared token override GPTBot, ClaudeBot or Google-Extended rules?
No. Company-specific rules keep working. Google applies the shared opt-out only when Google-Extended is not named, Anthropic says rules naming ClaudeBot take priority, and the draft lets providers treat their own tokens as more specific than the shared one.