OpenAI, Google, and two other tech giants unify AI training refusal rules, eliminating the need for websites to set preferences individually.

Beating AI News Flash: OpenAI, Google, Microsoft, and Anthropic have reached an agreement: going forward, websites only need to add two rules to `robots.txt` — `User-Agent: __usage_ai-training` and `Disallow: /` — to simultaneously tell these companies not to use their content for AI training. Previously, websites had to set rules separately for crawlers such as GPTBot and ClaudeBot.

This set of rules targets only AI training; it does not prohibit crawlers from scraping websites altogether, nor does it necessarily affect AI citing website content in answers. It was proposed by an Anthropic engineer and is currently still a technical draft of the Internet Engineering Task Force (IETF).

Companies are not entirely uniform in how they enforce the rules. Microsoft allows websites to separately prohibit AI training while preserving the opportunity for content to be cited by AI search. Google plans to bind this rule with the existing Google-Extended setting, potentially also restricting Gemini from citing website content.

Anthropic has already begun adopting it, Microsoft plans for it to take effect in February 2027, Google plans to support it in 2027, and OpenAI will also roll it out in the coming months.

However, these rules cannot retract data already used for training, nor can they force AI companies to comply.