OpenAI will honor shared robots.txt signal for AI training preferences

OpenAI said it will honor a shared robots.txt signal for website operators to express whether their content should be used for foundational generative AI training. The voluntary mechanism is designed to let a publisher communicate the same content-use preference to multiple participating AI providers.

The company said it aims to implement the signal in the coming months and will later publish implementation details. OpenAI said the shared signal will not replace its existing robots.txt crawler controls, which will continue to apply.

The proposed specification defines a shared policy token, __usage_ai-training, for the User-Agent field in robots.txt. A site can use ordinary Allow and Disallow rules in that group to express path-specific training preferences without adding new robots.txt directives or syntax.

The shared token conveys a content-use preference; it does not control whether a crawler may retrieve a path. Provider-specific crawler product tokens continue to govern access, making the shared signal separate from the bot-by-bot rules that publishers use to manage crawler traffic.

Under the draft, the absence of a __usage_ai-training group means the site has expressed no preference. A wildcard User-Agent group must not be treated as matching the policy token. The draft also permits more specific proprietary preference tokens and private agreements to take precedence.

The document is an individual Internet-Draft and has no formal standing in the IETF standards process. The IETF AI Preferences working group is chartered to develop mechanisms for expressing how content may be collected and processed for AI, while technical enforcement is outside its scope.

OpenAI currently documents separate controls for OAI-SearchBot and GPTBot. OAI-SearchBot governs inclusion in ChatGPT search features, while blocking GPTBot indicates that crawled content should not be used to train OpenAI’s generative AI foundation models.