A single robots.txt line to block AI training: what the OpenAI-Google-Microsoft-Anthropic pledge does and doesn’t fix

The Verge reports that four of the largest AI developers have committed to respecting one shared signal in robots.txt, the file websites use to tell crawlers what they may access. Today a publisher that wants to keep its work out of AI training has to block each company’s crawlers separately and keep up as new ones appear. The new token would collapse all of that into a single line.

The proposal is moving through the Internet Engineering Task Force, the body that maintains many core web standards, and its author is a standards lead at Anthropic. Anthropic already honors it. Google and Microsoft plan to add support next year, and OpenAI says it will follow in the coming months. Meta said it supports the idea without committing to obey it.

A standards-track signal backed by the biggest labs beats a growing patchwork of crawler names that every small publisher must track. It lowers the cost of saying no, and that matters most for outlets without a technical team. Robots.txt has long governed indexing and archiving, but it was never designed with AI training in mind. Publishers have answered the surge in AI crawler traffic with blocking, licensing deals and lawsuits. This is the first time the major labs have rallied around one shared mechanism.

Microsoft’s plan is the most interesting detail. It intends to offer separate controls for training and for grounding, the practice of pointing a model at a source to support its answers, and says sites will remain in Bing search whichever they choose. Google already offers a similar split. The distinction matters economically. Training is a one-time use of a publisher’s work. Grounding is continuing use to answer questions that might otherwise have sent a reader to the publisher’s site.

The cost of that shift is already showing up in court. Unsealed filings in the New York Times-led case quoted a Microsoft executive calling AI scraping the “largest theft of labor in human history,” alongside internal data on collapsing click-through rates (our coverage).

This is still an opt-out. Content scraped before the signal existed is not coming back. Compliance is voluntary, and the timelines run into next year. The most telling line in the story is Meta’s: supportive words, no commitment. A standard only works if the companies that matter obey it, and the ones that don’t sign on are exactly the ones a publisher should worry about.

It also says nothing about payment. Efforts like the Really Simple Licensing standard and Cloudflare’s default blocking exist because many publishers want a price, not only a door they can close. A clean opt-out could even help AI companies argue that anyone who didn’t use it consented. The fair-use fight continues on a separate track, with OpenAI making its case publicly (our analysis of OpenAI’s fair-use argument).

Take the win and keep pushing. Useful AI depends on good information, and good information depends on publishers who can afford to keep producing it. A shared no is a floor, not a business model. More in our regulation coverage.