AI giants cave: one line will stop them training on your site

On October 30, OpenAI, Google, Microsoft and Anthropic backed a shared signal for website owners. To keep your content out of AI training, you no longer have to list every crawler in robots.txt one by one. A single group with a common "policy token" will be understood by every participating company at once.

It looks like a normal robots.txt entry: the User-Agent line says __usage_ai-training, followed by Disallow: /. You can keep exceptions, for example leaving only a press kit section open for training. There are no new directives or syntax, so existing parsers simply ignore the group.

The proposal comes from Kevin Kelley, a technical lead for standards at Anthropic, through the AIPREF working group at the IETF, the body that writes internet standards. For now it is a draft, not an adopted standard.

Timelines differ. Anthropic already honors the signal. Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the coming months. Google will treat an opt-out as a disallow for Google-Extended unless that crawler is listed separately. Microsoft will add finer controls, letting sites opt out of training and of AI grounding separately, while staying in Bing search results either way. Existing per-crawler rules keep working.

The reason is obvious: publishers have been fighting AI crawlers that scrape their sites. Some charge bots for access, and Cloudflare blocks them by default. Now the biggest players have handed out a single "keep out" switch themselves.

Verdict: compliance is voluntary, but one line instead of a dozen rules is a real win for anyone who runs a website.