One robots.txt line to opt out of AI training: what the shared signal does and doesn't do

Google, Microsoft, OpenAI and Anthropic will honour one shared AI training opt-out in robots.txt, The Next Web reported on 31 October. It lets a website state a preference against generative AI training with a single rule group. The signal comes from an IETF draft by Kevin Kelley of Anthropic, published on 30 October, and Kelley announced it on the IETF's AI preferences mailing list. The announcements are real, but the signal is a draft and a preference.

How the shared line works

A site adds a group for a new policy token, __usage_ai-training. Search Engine Journal reports that Kelley's draft adds no new directive to robots.txt: the group is written as it always has been, using Allow and Disallow. The draft's example, which blocks a whole site except its press kit, reads: User-Agent: __usage_ai-training, then Disallow: /, then Allow: /press-kit/. The draft says this expresses a preference against AI training for all paths except those under /press-kit/, for which it expresses a preference in favour.

The token takes its meaning from the IETF working group's vocabulary draft, which defines AI training as using content to modify a generative AI model's learned parameters. The draft says training models used exclusively for non-generative tasks is not included, even if the model is capable of generative tasks.

What the signal does not do

The draft says the token states a preference about how sites want their content used. It is not a directive for crawling behaviour. Under the draft, existing rules for all crawlers don't count: a wildcard group (User-Agent: *) must not be read as matching the token, and a file with no __usage_ai-training group expresses no preference. Search Engine Journal reports that Kelley states this is true of every robots.txt file currently published. The draft also says private agreements may override preferences expressed this way.

The draft is an individual submission, and the IETF's datatracker says the IETF has not endorsed it and it has no formal standing in the standards process. Search Engine Journal adds that the tracker lists it as an individual draft, which does not represent official IETF standardisation activity. The draft expires on 3 May 2027, and Search Engine Journal reports that this holds unless a new version replaces it and that Kelley plans to propose his design as a change to the Content-Usage draft.

Who starts when

Based on the companies' own timelines, Search Engine Journal says none of Google, Microsoft and OpenAI currently supports the line. Anthropic has already started work on it, Kelley wrote, and he said it would complement Anthropic's existing crawler controls rather than replace them. "It'll take us some time to add support across our services and work through the inevitable complications and edge cases that I'm sure we'll discover," Kelley wrote, according to The Next Web. Search Engine Journal reports that as of 30 October there was no posting in Anthropic's newsroom about the signal, and that Anthropic's crawler-help page, checked that day, listed three crawl control options: ClaudeBot, Claude-User and Claude-SearchBot.

OpenAI said it aims to implement the signal "in the coming months", and its existing crawler controls, such as GPTBot, will still apply, it said. Search Engine Journal reports that neither OpenAI's blog post nor its crawler documentation, checked on 30 October, states what would happen when a site has a shared-line rule and a GPTBot rule that conflict.

Google: tied to Google-Extended

Google said it aims to support the signal in 2027, and its crawler documentation says early 2027. Google's post, by John Mueller, says that if a site opts out with the token and does not mention Google-Extended, Google plans to interpret that "as if the publisher had disallowed Google-Extended."

Search Engine Journal, reporting Google's documentation, says Google-Extended controls whether content can be used to train future Gemini models and whether it can be used for grounding in Gemini Apps and on Vertex AI. Google's post adds that sites that specifically allow Google-Extended are eligible to appear in Gemini Apps responses through grounding. The same documentation, as Search Engine Journal reports it, says Google-Extended doesn't affect a site's inclusion in Google Search. Search Engine Journal adds that AI Overviews and AI Mode have a separate opt-out, the Search generative AI control in Search Console.

Microsoft: replacing controls sites already use

Microsoft will honour the new preferences from 1 February 2027, and they will replace its NoArchive and NoCache preferences for training and grounding. Microsoft's own blog post says it was the first company to provide publisher controls for AI training and grounding, in 2023, with those tags. The post is by Krishna Madhavan, a principal product manager at Microsoft Bing, and, according to Search Engine Journal, adopts the shared line for training and adds a separate AI Grounding preference.

The grounding setting, for when an AI answer draws on a live web page, is page-level. Search Engine Journal says a new option called DisallowExtended allows grounding in Microsoft-created tools like Copilot and Bing's AI Mode but not through Microsoft's grounding services used by other businesses (Web IQ). Microsoft said either restriction will limit visibility of a brand in AI answers and will have no effect on its ability to be listed in Bing search results. It also said the new preferences "do not establish licensing or payment arrangements", as The Next Web reported.

Search Engine Journal reports that Microsoft will stop recognising the older tags as training or grounding options after 1 March 2027. It adds that sites using the old tags that want to stay out of grounding also need to record a temporary preference in Bing Webmaster Tools, a setting that opens on 15 December and is available only to sites still using those tags on that date.

What it means for site owners

Search Engine Journal says one robots.txt group could replace a stack of per-company training rules once each company reads it, but that an opt-out through the shared line won't reach the same uses at Google and Microsoft. It also says Microsoft's is the only announcement that replaces controls sites already use. According to Google, OpenAI and Kelley, existing controls will continue to work, so Search Engine Journal says sites that use NoArchive or NoCache to limit how Copilot uses their content have things they need to do first. On the evidence of these announcements, the shared line is a simpler way to state a training preference, not a single switch that works the same everywhere.

The Next Web, citing The Verge, reported that Meta communications head Andy Stone voiced support for a common signal on X, but did not say whether Meta would honour it. Google said the proposal is consistent with the EU's Code of Practice for general-purpose AI under the AI Act, and The Next Web says the code recognises robots.txt as a main way for publishers to state machine-readable preferences.

Sources

Constantin, A. M. (2026) 'Google, Microsoft, OpenAI and Anthropic back one AI training opt-out', TNW | Artificial-intelligence, 31 October 2026, available at: https://thenextweb.com/news/ai-training-opt-out-robots-txt-google-openai-microsoft-anthropic (Accessed: 8 November 2026).

IETF Datatracker (n.d.) 'Machine-Readable Training Preference Signal', IETF Datatracker, available at: https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/ (Accessed: 8 November 2026).

Microsoft Bing Blog (2026) 'Your content, your choice: clearer preferences for training and grounding', Microsoft Bing Blog, 30 October 2026, available at: https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/ (Accessed: 8 November 2026).

Southern, M. G. (2026) 'Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out', Search Engine Journal, 1 November 2026, available at: https://www.searchenginejournal.com/google-microsoft-openai-shared-ai-training-opt-out/595387/ (Accessed: 8 November 2026).

Picture credits

• Top picture: Microsoft says it will honour the new preferences from 1 February 2027. Image: Building 92 of Microsoft Redmond Campus, by Jiaqian AirplaneFan, via Wikimedia Commons (CC BY 3.0)