# OpenAI, Google, Microsoft and Anthropic back a shared robots.txt opt-out for AI scraping

published: 2026-10-30T22:05:35.321Z
topic: search
source: https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out
url: https://aiagentnews.io/stories/openai-google-microsoft-and-anthropic-back-a-shared-robots-txt-opt-out-for-ai

OpenAI, Google, Microsoft and Anthropic say they will honor a shared signal that lets publishers set AI training preferences once across multiple AI companies. The signal lives in robots.txt, and Anthropic has already adopted it while Google, Microsoft and OpenAI have set later timelines.

OpenAI, Google, Microsoft and Anthropic say they will adopt a shared opt-out signal that lets publishers apply their AI training preferences across several AI companies instead of setting them one by one. The 'policy token' was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force. It sits in a site's robots.txt file, which already tells crawlers what they may index, archive or use for AI training.

Anthropic has already adopted the signal. Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the coming months. Each company says it will restrict its AI crawlers from accessing a site that carries the token. Meta's communications head, Andy Stone, voiced support for a common signal but did not say whether Meta will honor it.

Microsoft says it will offer granular controls that let publishers opt out of AI training or AI grounding, the process of pointing a model at a verified source to reduce hallucinations. Microsoft adds that websites will still appear in Bing search results regardless of those preferences. Google already offers controls over whether content appears in AI search features or is used for training, without affecting search visibility.

## Why it matters if you build agents

Agent builders that crawl or retrieve web content will need to check robots.txt for the token, but support varies by provider and arrives at different times. Per-provider settings may no longer be the only place a site states its AI preferences. Because Microsoft separates training from grounding, a site may allow one use and block the other.