OpenAI, Google, and Microsoft back one AI-training robots.txt signal

On Oct. 30, OpenAI, Google, and Microsoft said they will honor a shared robots.txt signal for generative-AI training. OpenAI aims to implement it in the coming months and keep its existing crawler rules, Google aims to support it in 2027 and to treat a training opt-out as a Google-Extended disallow unless that control is set explicitly, and Microsoft said it will honor User-Agent __usage_ai-training from Feb. 1, 2027 while adding separate page-level training and grounding controls. That token is described in an individual Internet-Draft dated Oct. 30, which the IETF has not adopted as a standard.