Big AI Firms Adopt a Shared Opt-Out Signal for Training

On October 30, OpenAI said it will start honoring a new "policy token" signal placed in a site's robots.txt file. Robots.txt is a plain text file that tells web crawlers what they can and cannot do. The new signal lets sites state their stance on AI training use of their content in one place, instead of spelling out a separate policy for every bot. Kevin Kelly, a standards engineer at Anthropic (maker of the Opus and Fable language models), first proposed the idea inside a working group of the Internet Engineering Task Force (IETF), the body that develops open internet standards. According to The Verge, OpenAI, Google, Microsoft, and Anthropic have committed to honoring this signal.

The change makes life a little easier for people who publish content online. Instead of writing a separate policy for every bot, they can now use one signal. But it is voluntary, and these companies carry no legal obligation to follow it. So it is worth exactly as much as a big corporation's promise usually is.