What the shared AI training opt-out means for brand websites

Google and Microsoft said on 30 October 2026 that they will honour one shared robots.txt token, __usage_ai-training, for opting content out of AI training, with Microsoft starting on 1 February 2027 and Google in early 2027. Brand sites now get one switch for training, and Microsoft has added a separate one for AI answers.

Key takeaways

• Google’s Search Central blog said on 30 October 2026 that it aims to support a shared AI training control in robots.txt in 2027, and its crawler documentation says early 2027.
• Microsoft Bing will honour the shared AI Training preference, plus a new page-level AI Grounding preference, as it crawls from 1 February 2027.
• The token comes from an individual IETF Internet-Draft by Kevin Kelley of Anthropic, dated 30 October 2026, which expires on 3 May 2027 and has no formal IETF standing.
• Rules under User-Agent: * do not count as a training preference, so a site with no group naming the token has expressed no preference at all.
• Sites that use Bing’s NoArchive or NoCache tags must publish the new preferences before 1 March 2027 to keep their restrictions.

The token Google and Microsoft signed up to

The mechanism is an ordinary robots.txt group. Instead of naming a crawler, the User-Agent line names a policy token, and the Disallow and Allow lines say which paths may or may not be used for training.

User-Agent: __usage_ai-training
Disallow: /research/

Kevin Kelley of Anthropic published the Internet-Draft on 30 October, and in his message to the IETF list he wrote that “Anthropic has already started implementing support for this draft.” The draft says the token “is not a directive for crawling behavior”, so a crawler still follows the rules written for its own name. Blocking a crawler and stating a training preference stay two separate jobs.

John Mueller announced Google’s plan on the Search Central blog. When a site uses the token to opt out and doesn’t mention Google-Extended, “Google plans to interpret that as if the publisher had disallowed Google-Extended.” Google’s crawler documentation puts support in early 2027 and repeats that Google-Extended doesn’t affect inclusion or ranking in Google Search. Barry Schwartz at Search Engine Roundtable reports that OpenAI has taken a similar step.

Why does Bing split training from grounding?

Microsoft went further than Google. Krishna Madhavan’s post on the Bing blog adopts the shared AI Training preference, extends it to single pages through a meta tag or HTTP header, and adds an AI Grounding preference that governs whether a page can inform AI-generated answers in Bing, Copilot and Microsoft 365. Grounding is set page by page only, with three values. Allow is the default, Disallow keeps the page out of generated answers, and DisallowExtended keeps it in Microsoft’s own products while excluding the grounding services Microsoft sells to third parties, such as Web IQ and grounding through Foundry.

Microsoft’s post warns that choosing DisallowExtended or Disallow “will significantly reduce your content’s visibility in AI-generated experiences”, while leaving eligibility for classic Bing results untouched.

Which control does what?

Where an opt-out costs visibility

Google’s post attaches a condition to the shared token, saying sites “that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps”, and a token-only opt-out counts as a Google-Extended disallow. A brand that adds one blanket training block could therefore drop out of Gemini answers without meaning to. Google ties Gemini Apps grounding to the same Google-Extended Allow that permits Gemini training, so on any one path a site can’t keep Google out of training and stay eligible for that grounding. Keeping the opt-out to the protected paths leaves the rest of the site eligible for both.

The draft also says a User-Agent: * group never matches the token, so the familiar catch-all rules say nothing about training. Existing crawler blocks keep working too, so a site that disallowed GPTBot or Meta-ExternalAgent after reading our piece on Meta’s crawling terms keeps those blocks whatever it writes for the new token.

Should a brand block AI training?

For most brands, no. Product pages, pricing, comparisons and category explainers exist to be read and repeated, and a model that learned your positioning is more likely to describe you accurately when it answers without searching. The SISTRIX citation study and the citation moved off the page cover what earns those pages a citation once a model goes looking.

The exceptions are content you sell or license. Paid research, member-only guides, licensed photography and customer data deserve a path-level Disallow, and the draft’s path matching lets you protect /research/ while leaving /products/ open.

Our view at Manual Focus is that brand and commercial sites should allow AI training by default, disallow it only on the paths that carry content they sell or license, and leave grounding allowed on every page that should appear in AI answers, because Microsoft says grounding restrictions cut visibility and Google ties Gemini grounding to Google-Extended. SEO, legal and brand teams should write that policy down together, since the person editing robots.txt rarely knows which content is licensed.

Five settings to fix before 1 March

1. Find your old Bing tags. Search templates and response headers for NoArchive and NoCache. If you use them, decide your training and grounding preferences now, and if you want to keep withholding pages from grounding, record a temporary preference in Bing Webmaster Tools once it opens on 15 December 2026.
2. Map which paths carry licensed content. List the directories with paid, gated or third-party content and draft path-level __usage_ai-training Disallow rules for those alone. The site audit and refresh playbook gives a structure for that inventory.
3. Keep Google opt-outs to the protected paths. An explicit Google-Extended Allow lets Google train on those paths as well as ground Gemini answers in them, so don’t add a site-wide one to undo a training block. Disallow the token, or Google-Extended, only on the paths from step 2 and leave everything else allowed.
4. Leave grounding on commercial pages at Allow. Reserve the Bingbot ai-grounding Disallow for pages that should never appear in an AI answer, such as account or legal pages.
5. Test the file after each change. A malformed group can block Googlebot or Bingbot from crawling. Treat robots.txt as brand infrastructure with an owner and a change log, as the brand guardrails as code playbook suggests for other brand rules.

Still unsettled

The draft is an individual submission, and Nate Hake told Search Engine Roundtable that “some significant areas of dispute remain” at the IETF, including an exemption for user-initiated actions. OpenAI’s timing isn’t in either the Google or the Microsoft announcement. Neither company says a new opt-out removes content already gathered into existing datasets or models, and Microsoft’s own post says the preferences cover “future AI generative model training”. Google’s post also doesn’t say how AI Overviews and AI Mode, which run inside Google Search, relate to the token. Microsoft says no setting guarantees inclusion, traffic or payment.

Further reading

• What Meta’s new crawling terms mean for UK advertisers, on which Meta crawlers to block and which to keep
• What Google’s new AI Mode links mean for brand websites, on how Google now sends AI Mode users to official sites
• The citation moved off the page, on why retrievability now matters more than rank
• Reddit plans a logged-in-only site, on another owner tightening who can read its content

If you want help setting a training and grounding policy for your site, our services page explains how we work.

I run marketing for [brand], [one line on what you sell] at [website], selling
mainly in [market].

We want to opt out of AI training only on the parts of our site that carry
content we sell, license or keep behind a login, and leave everything else
open. The shared robots.txt token for this is __usage_ai-training, and it works
by path with ordinary Disallow and Allow lines.

Attached are:
- our sitemap or a crawl export listing every URL
- a short note on what we sell or license (paid reports, member content,
  licensed images, customer data and so on)

Group the URLs by directory. For each directory, tell me whether it looks like
it holds sold, licensed or gated content, and why.

Give me a table with the columns: Directory | Example URLs | Number of URLs |
Sold, licensed or gated? (yes, no, unsure) | Reason.

Then draft the robots.txt group for the token that disallows only the
directories marked yes. List every directory marked unsure separately so a
person can check it against our contracts. Don't guess what a page contains
from its URL alone if you can't see it; mark it unsure.

Microsoft Bing is replacing its NoArchive and NoCache tags with new AI Training
and AI Grounding preferences, and will stop reading the old tags as AI
preferences after 1 March 2027.

Attached are our page templates and a sample of HTTP response headers from
[website].

Find every place where a NoArchive or NoCache instruction appears, in a robots
meta tag or an X-Robots-Tag header.

Give me a table with the columns: Template or URL | Where it appears (meta tag
or header) | Exact instruction | Which page types use this template.

Then list the questions our team has to answer for each one before 1 March
2027: do we want these pages used for AI training, and do we want them used to
inform AI answers. If a template includes others you can't see, or a header
sample is incomplete, say so rather than assuming it's clean.

Set this up as a check we run every time someone edits our robots.txt.

Attached are our current robots.txt and the proposed new version for
[website].

For each file, explain in plain English what each named crawler group and the
__usage_ai-training token allows and disallows, and which paths are affected.
Remember that a User-Agent: * group does not count as a training preference
for the token, that Google plans to read a token-only opt-out as a
Google-Extended disallow, and that an explicit Google-Extended Allow lets
Google train on the allowed paths.

Give me a table with the columns: Group | Current rules | Proposed rules |
What changes in plain English.

Then flag, at the top, anything in the proposed file that would stop Googlebot
or Bingbot crawling pages we want in search, any syntax that looks malformed,
and any Google-Extended Allow that covers a path where we opted out of
training. If you can't tell how a particular crawler reads a rule, say so
rather than guessing.