
Microsoft Split AI Training From Citation. Google Called That Free-Riding.
A shared robots.txt line will let a shop refuse AI training at several providers at once. Microsoft keeps citation a separate choice; Google plans to read the line as a Google-Extended block, which also drops the shop from Gemini's grounded answers, a split Google called free-riding in its response to the UK regulator.
Neritus Vale
One robots.txt group will soon let a shop refuse AI training at four of the biggest AI companies. At Microsoft, OpenAI and Anthropic, citation stays a separate choice, open by default. At Google, the same line also closes Gemini’s grounded answers, because Google plans to treat it as a Google-Extended block.
The line comes from draft-kelley-aipref-training-preference-signal-00, an Internet-Draft posted on 30 October by Kevin Kelley of Anthropic. Its “policy token”, __usage_ai-training, sits where a crawler’s name would go. The datatracker lists it as an individual draft, not an IETF AIPREF working-group document. It borrows that group’s definition of training: using an asset to “modify the learned parameters of a generative AI model”.
Grounding is not training under that definition: Google’s crawler documentation describes it as “providing content from the Google Search index to the model at prompt time”. Yet the same page gives Google-Extended both jobs: Gemini training, and grounding in Gemini Apps and Vertex AI. John Mueller’s Search Central post says Google plans to read a token opt-out that omits Google-Extended “as if the publisher had disallowed” it. A preference about training arrives at Google as a refusal to be cited by Gemini.
Google’s post spells out the trade: sites that “specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” The Gemini app “is now at 1 billion users”, Sundar Pichai said on Alphabet’s third-quarter call. Its shopping features, described on a March 2026 page for Canada, offer “shoppable product listings” and “places to buy”, and are “powered by the Shopping Graph”. Neither the post nor the crawler page says whether the Google-Extended switch reaches those listings.
Google’s Search Console page for its Search control adds that the control “doesn’t override publishers’ other choices to participate in particular services, such as Merchant Center”. That leaves open that Gemini’s product listings come from merchant feeds rather than from grounded pages. A shop would need to know which before it decides what the switch costs.
Microsoft announced the opposite design the same day. Its Bing post adopts the token for training and adds a separate AI Grounding preference, which is a page-level setting only. The old tags, NoArchive and NoCache, “treated training and grounding as a single decision”, its FAQ says. Microsoft’s own example sets AI Training to Disallow and AI Grounding to Allow; because grounding has no robots.txt form, a shop that wants to change it must tag its pages.
OpenAI and Anthropic already keep the two apart by crawler. OpenAI’s bot documentation lets a site allow OAI-SearchBot “to appear in search results while disallowing GPTBot”, and the company aims to implement the shared signal “in the coming months”. Anthropic, which Kelley says “has already started implementing support”, separates ClaudeBot, its training crawler, from Claude-SearchBot and Claude-User. Meta’s Andy Stone welcomed “a common signal” on X without saying Meta would honour it.
| Company | Shared opt-out removes | Search/retrieval crawler or tag | Starts |
|---|---|---|---|
| Anthropic | Training | Claude-SearchBot, Claude-User | Under way |
| OpenAI | Training | OAI-SearchBot | Coming months |
| Microsoft | Training | Page-level grounding tag | 1 Feb 2027, on recrawl |
| Gemini training and grounding | Search generative AI control (Search Console) | Early 2027 | |
| Meta | Not committed | Not stated | No date |
Google has already split the two inside Search. Its Search generative AI control, a Search Console setting rolled out worldwide by 31 August, lets a site leave AI Overviews and AI Mode but “doesn’t affect AI training”. The Verge notes that Google “previously implemented similar controls”. Under Google’s plan, a shop that writes only the shared line would stay eligible for AI Mode citations and lose Gemini’s.
The Verge also treated the token as a crawler block. It reported that the companies “will restrict their AI crawlers from accessing a site”. The draft says the token “is not a directive for crawling behavior”, though crawlers may skip pages they cannot use. Googlebot keeps fetching the catalogue either way: Google-Extended, Google’s documentation says, works “in a control capacity”.
The Gemini gap has a regulatory edge. The Competition and Markets Authority’s October 2025 designation of Google in general search covered AI Overviews and AI Mode and confirmed that “Google’s Gemini AI assistant is not in scope”, a position it said would be kept under review. Google began testing the Search toggle with UK sites on 3 June, the day the CMA imposed a conduct requirement for opt-outs of that kind. Gemini got no toggle of its own.
The configuration a retailer wants, cited but not trained on, already has a name in Google’s submissions to that regulator: free-riding.
Google argued, as the CMA’s final decision records, that separate training and grounding opt-outs outside general search created “a ‘free-riding’ concern”. The decision’s summary of Google’s response adds that publishers “could opt in to grounding to maintain traffic, but opt out of the training which makes generative AI products possible.” It uses “publisher” for “all parties that make content available on the web”, which takes in a shop. The CMA had “not seen evidence that further separation of the Google-Extended control is necessary” to address its concerns, which were Google’s advantage from general search and clarity over the role of Google-Extended. It left the switch whole. Google’s October plan routes the new shared line through that same control.
Under the draft, a refusal binds only the robots.txt “in effect when the asset was retrieved”. A shop that writes the line in 2027 withholds what is fetched afterwards, not the seasons already crawled.
The strongest objection is that none of this binds Google until it ships. Its post speaks only of plans, and Microsoft has just shown that a bundled switch can be split. For the thesis to fail, Google would have to divide Google-Extended into training and grounding before it reads the token in early 2027. Nothing on the record points that way: Google argued against that kind of division to a regulator this year, and its plan sends the shared line into the bundle.
A retailer can still qualify for Gemini’s citations by adding an explicit Google-Extended allow, as Google’s post invites. The price is Google’s training: Google applies the shared token only where Google-Extended is not explicitly specified. The choice Google leaves a shop is the one it defended before the CMA: cited and trained on, or neither.