AI companies plan a shared robots.txt training preference

Google, Microsoft, OpenAI and Anthropic have backed a proposed robots.txt signal that would let website operators express an AI-training preference for multiple providers with one rule. The signal concerns future training use, does not itself block crawling, and providers plan to support it on different timelines.

• The proposed token, __usage_ai-training, would let site operators state preferences for training use across participating providers instead of maintaining separate rules for each provider.· 2 sources
• The draft describes a preference about content use, not a crawler-blocking instruction; it also cannot reverse scraping that has already occurred.· 2 sources
• Anthropic says it has begun support; OpenAI plans implementation in the coming months, while Google and Microsoft have named 2027 timelines.· 2 sources
• The proposal is an individual Internet-Draft and has not been endorsed by the IETF or given formal standing in its standards process.· 2 sources
• Microsoft and Google have said opting out of training may affect a publisher’s visibility in AI services; the new signal does not itself prevent crawling.· 2 sources