On 30 October 2026, Google, Microsoft, OpenAI and Anthropic said they will honour a single line in your website's robots.txt file that tells AI companies whether your content may be used to train their models. Instead of listing every AI crawler by name, you can state your AI training preference once, using a shared token called __usage_ai-training.
For most business websites, the practical answer is simple: this rule controls training, not whether you show up in Google, Bing, ChatGPT search or AI Overviews. You can opt out of training and still be found. Below we explain what changed, the exact lines to add, the dates each company has given, and how to decide what is right for your business.
What was announced
The rule comes from an Internet-Draft titled Machine-Readable Training Preference Signal, published at the IETF on 30 October 2026 by Kevin Kelley of Anthropic. It is discussed in the IETF's AI Preferences (aipref) working group. It is an individual draft, not a finished standard, and it is labelled as expiring on 3 May 2027 unless it is updated.
The same day, the big AI providers announced support:
- Google published Supporting a shared AI training control for websites on the Search Central blog. Its common crawlers documentation now says Google intends to support the signal in early 2027.
- Microsoft Bing announced new training and grounding preferences in Your content, your choice, and says they begin taking effect on 1 February 2027.
- OpenAI and Anthropic also announced support, as reported by Search Engine Roundtable and Coywolf. Neither company had given a firm switch-on date in the sources we checked.
Search Engine Roundtable also notes that people involved in the IETF work say some areas of the proposal are still disputed, so details may change before it becomes a standard.
How the new robots.txt rule works
The draft does not invent new syntax. It reuses normal robots.txt groups, but the User-Agent line names a "policy token" instead of a real bot. A Disallow means "please do not train on this", and an Allow means "training is fine".
To opt your whole site out of AI training:
User-Agent: __usage_ai-training
Disallow: /
To opt out everywhere except one folder you are happy to share, such as a press kit:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
A few details from the draft are worth knowing:
- A wildcard group (
User-Agent: *) does not count as a training preference. You must name the token. - If no group names the token, you have expressed no preference.
- The token is not a crawling instruction. Your existing rules for Googlebot, Bingbot and other crawlers keep their normal meaning.
- "Training" means using content to change the learned parameters of a generative AI model. Models that only classify, rank or score are outside its scope.
Google adds one more rule of its own: if you use the shared token to opt out and do not mention Google-Extended separately, Google will treat your site as if you had disallowed Google-Extended.
What this does not do
This is where many site owners get confused, so it is worth being precise.
It does not remove you from search
Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Microsoft says its new preferences "do not change eligibility for traditional Bing search results". Opting out of training will not, on its own, hurt your rankings.
It does not control AI answers and AI Overviews
Training and answering questions are different uses. When an assistant reads your page to answer a question right now, that is often called grounding. The shared token only covers training. Each company handles answers with separate controls:
| Company | Training control | Search and AI answer controls |
|---|---|---|
Google-Extended token (and the shared token from early 2027) |
The Search generative AI control in Search Console for AI Overviews, AI Mode and Discover | |
| Microsoft Bing | Shared token in robots.txt, or ai-training:disallow per page |
Per-page ai-grounding setting: Allow, DisallowExtended or Disallow |
| OpenAI | GPTBot |
OAI-SearchBot for ChatGPT search; ChatGPT-User for user-requested visits |
| Anthropic | ClaudeBot |
Claude-SearchBot for search; Claude-User for user-requested visits |
Google's Search Console control is separate again. According to Google's help page, excluding your site there stops your content appearing in AI Overviews, AI Mode and generative features in Discover, it is "not used as a ranking or inclusion signal" for the rest of Search, and it does not affect AI training.
Microsoft's options are the most detailed. Its grounding preference is set per page, with a middle option, DisallowExtended, that limits grounding to Microsoft's own products such as Copilot and Microsoft 365. Microsoft warns that choosing DisallowExtended or Disallow "will significantly reduce your content's visibility in AI-generated experiences."
It is not a lock
robots.txt is a request, not a security barrier. Reputable companies follow it; scrapers that ignore it will not. If you need to actually stop unwanted bots, that is a job for a firewall or bot-management service at your host or CDN.
Key dates for your calendar
| Date | What happens |
|---|---|
| 30 Oct 2026 | Draft published; Google, Microsoft, OpenAI and Anthropic announce support |
| 15 Dec 2026 | Bing opens temporary grounding preferences in Bing Webmaster Tools for sites using NoArchive or NoCache |
| Early 2027 | Google intends to start supporting the shared token |
| 1 Feb 2027 | Bing's new training and grounding preferences begin taking effect as it recrawls |
| 1 Mar 2027 | Bing stops reading NoArchive and NoCache as AI preferences |
| 3 May 2027 | Current version of the IETF draft expires unless updated |
If your site currently relies on NoArchive or NoCache meta tags to keep content out of Bing's AI features, the March 2027 change matters to you: those tags will no longer be read that way, so you will need to publish the new preferences instead.
Should your business opt out of AI training?
There is no single right answer. It depends on what your content is worth and how you win customers.
Consider opting out if:
- Your content is the product, such as paid courses, research, original photography or detailed guides you sell.
- You publish large amounts of original writing that competitors could benefit from.
- You have licensing deals or plan to negotiate them.
Consider allowing training if:
- Your website exists mainly to win enquiries, and you want AI tools to know your brand, services and location.
- Your pages are mostly service descriptions, contact details and FAQs that you want repeated as widely as possible.
- You care more about being mentioned than about controlling reuse.
For a typical service business, such as a clinic, a law firm, an estate agent or an agency like ours, the biggest risk is usually not training. It is accidentally blocking the search and answer bots that send customers. We regularly see security plugins and firewall presets that block AI user agents by default without the owner knowing. As we covered in our guide to SEO for Google AI Overviews, being cited in AI answers is becoming a real source of visibility.
A five-step checklist
- Read your current robots.txt. Visit
yourdomain.com/robots.txtand note any rules for GPTBot, ClaudeBot, Google-Extended or other AI agents. - Decide training and answers separately. Write down one decision for training and one for search and AI answers. They do not have to match.
- Add the shared token if you want to opt out of training. Keep any company-specific rules you already have; all four companies say their own tokens continue to work.
- Check your firewall and plugins. Make sure nothing is blocking
OAI-SearchBot,Claude-SearchBot, Googlebot or Bingbot if you want to appear in search and AI answers. - Review Search Console and Bing Webmaster Tools. Confirm the Search generative AI setting in Google Search Console, and plan Bing's page-level preferences before the 1 March 2027 deadline if they apply to you.
If you would like a second pair of eyes, this is part of the technical checks in our SEO service. Shops selling online should also read our guide to agentic commerce, since AI shopping agents rely on the same crawlers.
Frequently asked questions
What is the __usage_ai-training token?
It is a shared policy token proposed in an IETF draft published on 30 October 2026. You put it in the User-Agent line of your robots.txt file with Disallow or Allow rules to state whether your content may be used to train generative AI models.
Will opting out of AI training hurt my Google rankings?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal, and Microsoft says its new preferences do not change eligibility for traditional Bing results.
Does opting out of training remove my site from ChatGPT or AI Overviews?
No. The token covers training only. Appearing in AI answers is controlled separately, for example by OAI-SearchBot for ChatGPT search, the Search generative AI control in Google Search Console, and Bing's page-level grounding setting.
When does the new rule start working?
Google says early 2027 and Microsoft says 1 February 2027. OpenAI and Anthropic have announced support but, in the sources we checked, had not published firm dates. Adding the line now does no harm.
The bottom line
The shared AI training opt-out finally gives website owners one clear place to say "please do not train on my content". It is useful, but it is not a search setting and not a security tool. Make two separate decisions, one for training and one for search and AI answers, and check that nothing on your site is quietly blocking the bots that bring you customers.
Not sure what your site is currently allowing? Talk to Mistique Designs and we will review your robots.txt, crawler access and AI search visibility for you.