Google, Microsoft, OpenAI Back One Robots.txt AI Opt-Out

By
Google, Microsoft, OpenAI Back One Robots.txt AI Opt-Out
Google, Microsoft, OpenAI Back One Robots.txt AI Opt-Out
Affiliate disclosure: This post contains affiliate links. If you buy through them, I may earn a commission at no extra cost to you. Full disclosure

Google, Microsoft and OpenAI said on October 30 that they will honor a single robots.txt token that lets a website opt out of AI model training.

If you run a Shopify store, your product copy, spec tables and photos sit on public pages that every AI crawler can read. This token is the first one-line way to say no to training at several companies at once, and it is worth knowing about at Ecommerce Paradise because it does not decide whether ChatGPT, Gemini or Copilot recommend your products in shopping answers. Mixing those two things up is the expensive mistake.

I’ll cover what was announced and the dates attached to it, how opt-outs got this messy, what it means for a store selling $1,000-plus products as I teach in my guide to high-ticket dropshipping, and the checks I’d run before Bing’s March 1, 2027 deadline.

Know which pages earn your organic traffic before you touch a crawler rule. Semrush shows you the pages and keywords that actually bring buyers, so you block or open the right paths. Check your top pages in Semrush →

Google, Microsoft and OpenAI Will Honor One Robots.txt AI Token

The mechanism is a policy token named __usage_ai-training. A site owner writes it into a robots.txt group the way they would write a crawler name, then lists the paths they do not want used for training. The example in the draft blocks a whole site except a press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The proposal is an Internet-Draft by Kevin Kelley of Anthropic, dated October 30, 2026, according to the IETF datatracker listing. It is an individual submission. The IETF has not endorsed it, and the draft expires May 3, 2027.

Each company committed on a different schedule, per The Next Web. PPC Land reported the same timeline.

Google says it intends to support the signal in early 2027, according to its crawler documentation. John Mueller, Google’s Search Advocate, said a site that uses the new token without mentioning Google-Extended will be treated as if it disallowed Google-Extended. Google also published a Search Central post titled Supporting a shared AI training control for websites.

Microsoft will honor the preferences from February 1, 2027. Krishna Madhavan, principal product manager for Bing, wrote that “publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work,” as quoted by TNW. Bing now offers three settings: AI Training, AI Grounding (whether your content can help generate AI answers) and Limited Grounding, which allows Microsoft’s own products but not its third-party grounding services, per Search Engine Roundtable.

OpenAI aims to implement the signal “in the coming months” and says its existing robots.txt controls stay in place. Kelley said Anthropic has started building support but that “it’ll take us some time to add support across our services,” with no launch date given.

Microsoft’s dates are the firm ones. December 15, 2026 is when temporary grounding controls appear in Bing Webmaster Tools for sites using the older NoArchive or NoCache directives. March 1, 2027 is the deadline for those sites to configure the new preferences if they want to keep their restrictions. September 30, 2027 is when the temporary preferences expire.

The signal is narrower than the headlines make it sound. It expresses a preference about how content is used. It is not a crawling order, so blocking a crawler still needs a rule that names that crawler. A wildcard User-agent: * group does not count, and a file with no __usage_ai-training group expresses no preference at all. Private licensing deals can override it, and the draft says nothing about content collected before you add the token.

Google’s post says Search inclusion is unaffected and that AI Overviews and AI Mode are governed separately by Googlebot rules and a Search Console control, per PPC Land’s reading. Meta, Amazon and Apple made no commitments, according to PPC Land. Meta’s communications head Andy Stone voiced support for a common signal on X but did not say whether Meta will honor it, per TNW.

Why Robots.txt Opt-Outs Needed One Shared Signal

Until now, a site that wanted out had to write separate rules for each company’s crawler, as TNW describes it. Google had Google-Extended. OpenAI had its own agents. Microsoft has used NoArchive and NoCache tags as AI controls since 2023, and those retire as AI controls after March 1, 2027.

Pressure on the old system has been building for months. Cloudflare told AI crawlers in July to pay or get blocked, and its CEO warned that bots could reach 1,000 times human web traffic, per TNW. I covered the EU privacy board’s move to give robots.txt weight in AI scraping disputes in my October 21 post on the EDPB, and Google says the shared token fits the EU’s Code of Practice for general-purpose AI, which recognizes robots.txt as a main way to state machine-readable preferences.

Compliance is the weak point. TollBit found about 15% of AI page fetchers in Europe reached disallowed URLs in August 2026, per PPC Land. HasData reported on October 8 that a Cloudflare change removed AI opt-outs from the robots.txt files of 118 sites, also per PPC Land. A rule you wrote can vanish without you touching it.

Google’s Gary Illyes added a warning on a podcast with Aneta Samkoff, per PPC Land: AI has made Googlebot impersonation worse, robots.txt is not a security control, and you should verify Googlebot by IP address and reverse DNS rather than by user-agent string.

The counterpoint

Nate Hake, a publisher in the IETF working group, warned on X that the work is unfinished and that some disputes remain open, per PPC Land. The commitments carry no enforcement mechanism, and OpenAI itself calls its participation voluntary.

Then there is the visibility trade. Microsoft says blocking grounding “will significantly reduce your content’s visibility in AI-generated experiences,” while insisting that Bing search rankings do not change. Cyrus Shepard posted data on LinkedIn from one site where removing it from AI Overviews and AI Mode left an estimated click-through rate of about 0.9% on those features, with AI impressions at about 8% of the site’s total, per PPC Land. His organic clicks also fell, which he called a “Search SERP Halo Effect.” Barry Schwartz noted the limits: one site, AI Overviews still showed on other results pages, and new content shifted average position.

What the AI Training Opt-Out Means for Your Product Pages

Everything below is my read, not reporting.

There are three separate switches, and the new token only touches one. Training is whether your pages help build the next model. Grounding is whether a live AI answer pulls from your page and cites it. Search inclusion is whether Google or Bing index you at all. The shared token covers training. Google’s version also takes grounding in the Gemini app and Vertex AI with it, since Google maps the token to Google-Extended and offers no grounding-only setting, per PPC Land. Mueller made the point that sites allowing Google-Extended can appear in Gemini app answers and get clicks.

For a high-ticket store, grounding is the switch that matters. A shopper comparing a $2,400 massage chair or a $6,000 generator asks an AI tool for a shortlist before they ever type a product name into Google. If your page is the source of the spec comparison, being cited is the new first impression. I looked at how fast the big retailers are moving on this in Walmart opening to AI shopping agents while Amazon blocks them.

Training is a different question: do you mind a model absorbing your work without sending anyone to your store? That depends on what the work is. If your product descriptions come straight from a supplier feed, the same words sit on a dozen competitor sites, and blocking training on them protects almost nothing. If you wrote original buying guides, shot your own photography or built comparison tables that took a week, those are assets worth fencing off.

Here is rough math, labeled hypothetical and built on Shepard’s single-site numbers. Say a store gets 100,000 Google impressions a month. If 8% of them come from AI features, that is 8,000 impressions. At a 0.9% click-through rate, that is about 72 clicks. At a 2% conversion rate on a $1,800 average order, those clicks are worth roughly $2,600 in monthly revenue. Small, but it ignores the halo effect Shepard reported, and one site is not a trend. Run your own numbers before you decide, because a store with a heavier AI footprint could land very differently.

Three scenarios, with thresholds:

Mostly supplier copy, little original content. Skip the opt-out and fix the content. Rewriting the top 20 product pages with your own specs, installation notes and photos does more for you than any robots.txt line, and my store audit write-up shows the kind of gaps I’d fix first.

Original guides and photography on a few paths. Use the token’s path rules to disallow training on those folders only, and leave product pages open to grounding. The draft supports path-level Allow and Disallow, which is the useful part for a store with a blog subfolder.

Worried about bot traffic and fraud rather than training. The token does nothing here. Retailers are working on agent identity instead, as in Home Depot asking AI shopping agents to prove who they are. Shopify’s Fall Edition added fraud rules and AI shopping controls on the platform side.

My read on timing: do nothing rash this week. Only Microsoft has a firm start date, Google says early 2027, and OpenAI and Anthropic have given none. A parser that does not recognize the token ignores it, per the draft, so adding it early costs nothing, but when Google starts honoring it you will lose Gemini grounding along with training. That is a decision to make on purpose, not by copy-paste from a blog post.

This is also one more layer of technical upkeep on a store that already juggles feeds, tracking and ad policies. Meta, for example, pushed its ad terms to November 20 and kept a crawl clause in them. If you would rather have someone run that whole technical stack while you focus on suppliers and sales, that is what my team does in our turnkey done-for-you high-ticket store service.

Want your store’s technical setup handled while you focus on sales? My team builds and runs high-ticket stores, including the crawler, feed and tracking details that keep showing up in the news. See the turnkey done-for-you service →

Robots.txt AI Opt-Out Checklist Before Bing’s March 1 Deadline

Five checks, in the order I’d do them.

  1. Open yourstore.com/robots.txt and save a copy today. On Shopify you edit it through the robots.txt.liquid template in your theme code, so confirm nothing in your apps or CDN has rewritten it, given the 118-site Cloudflare case HasData reported.
  2. Pull your top 20 organic landing pages and their rank history in Mangools before you change anything. If traffic moves after a robots.txt edit, you want a baseline to compare against.
  3. Sort your content into two piles: original (guides, photography, comparison tables) and supplier-fed. Only the first pile is worth a training opt-out, and the second is a rewrite project.
  4. Decide on Google separately from Bing. If you want Gemini app citations, do not use the shared token once Google supports it. For Bing, check Webmaster Tools after December 15 if you use NoArchive or NoCache, and set AI Training and AI Grounding on purpose before March 1, 2027.
  5. Verify bots by IP and reverse DNS instead of trusting user-agent strings, per Illyes, and check how AI referrals show up in your analytics. Related housekeeping: Google wants shared pages noindexed, and the same audit pass catches both.

Put December 15, February 1 and March 1 in your calendar now. If you are still choosing what to sell, start with my high-ticket niches list. Then build original content around it, using my guide to building a store brand.

Frequently Asked Questions

Does the new token stop ChatGPT or Gemini from recommending my products?
No. It covers training only, per the draft. Grounding, where a live answer cites your page, runs through company-specific controls, and Google’s version of the token also removes Gemini app grounding because it maps to Google-Extended.

Do I need to add it today?
Probably not. Microsoft honors it from February 1, 2027, Google says early 2027, and OpenAI says “in the coming months.” Adding it early is harmless, per the draft, as long as you accept the Google trade-off later.

Will a training opt-out hurt my Google rankings?
Google says Search inclusion is unaffected, and Microsoft says Bing rankings do not change, per PPC Land and TNW. Blocking grounding is where Microsoft warns visibility in AI experiences drops.

My descriptions come from a supplier. Is there anything to protect?
Not much, because the same text lives on every other store carrying that line. Your edge is original specs, photos and buying advice. My paid supplier directory helps you find suppliers that give you better source material to build from.

Is robots.txt a security tool?
No. Illyes said as much, per PPC Land, and impersonating bots ignore it. Verify crawlers by IP and reverse DNS, and read how AI Mode groups Google Shopping ads under one label if you want the paid side of the same shift.

Where do I start if my feed and product data are a mess?
Fix the data first. My Google Merchant Center checklist is the place to start. Then look at Shopify’s per-channel product titles, which let you tune what each channel sees.

Want to work through AI visibility decisions with other store owners and me? Inside the community you get the course library, member discussion and direct access to me. Start the 7-day free trial, then it is paid. Join the Skool community →

I’ll keep watching for Google’s and OpenAI’s implementation notes, because the exact wording of those docs will decide how much of this you can actually use. Subscribe to the YouTube channel for daily breakdowns. More breaking news coming through the day.

Not legal advice: the draft is a preference signal, not a legal instrument, and a lawyer should review anything involving licensing or privacy law.

Related Articles

If this was useful, these go deeper:

Share on Facebook
Post on X
Save
Share