Google, OpenAI, Anthropic and Microsoft Back One Switch to Keep Your Site Out of AI Training
A proposed robots.txt directive would let publishers block AI-training crawlers with one rule instead of one rule per company.
Google, OpenAI, Anthropic and Microsoft have committed to honouring a new setting that lets website owners say whether their content may be used to train AI. According to Engadget, the companies are adopting a proposal under consideration at the IETF, the body that sets internet standards. It was written by Anthropic standards lead Kevin Kelley.
Today a publisher who wants to opt out has to find each company’s crawler (the automatic program that visits pages) and write a separate rule for it. The proposal replaces that with one shared signal in the robots.txt file, the small text file at the root of a website that tells crawlers what they may visit. The example looks like this: a user agent called __usage_ai-training, with Disallow: / to block everything and, if wanted, Allow: /press-kit/ to permit one section.
Rollout differs by company. Anthropic has already started supporting it. Microsoft plans to start on February 1, 2027, Google in early 2027, and OpenAI has said “in the coming months.”
What’s behind it: For years the question “can AI companies train on my site?” had a messy answer, because every provider had its own rules and names. A shared label makes it easier for a small publisher to say no once and be done. Two limits are worth knowing. The directive applies only to AI training: it does not stop all crawling and does not remove pages from Google Search. Google says it will read the signal as a block for Google-Extended, the agent that collects material to train Gemini. And it cannot undo the past. Content that was already collected stays collected; the rule covers future crawling only. Also, robots.txt is a request that honest crawlers follow, not a lock.
What this means for you: If you run a blog, shop or portfolio site, ask whoever manages it whether the new rule will be added once it is final, or check your robots.txt settings in your website platform. If you want your work available to AI tools, you do not need to do anything. If you are only a reader, it is a small sign that the big labs are agreeing on basic etiquette, even if the details will take months to land.
Sources
Source: https://www.engadget.com/2303225/google-openai-anthropic-ai-crawling-controls/
Anthropic's Numbers Ahead of Its IPO: Revenue Up Twelvefold, Loss of $23 Billion
Reports on Anthropic's updated IPO filing show nearly $17 billion in quarterly revenue, a $23.3 billion net loss and $895 billion in compute commitments.
